[Vmware] Improve listing of Vmware Datacenter VMs for migration to KVM #10770

nvazquez · 2025-04-24T09:55:06Z

Description

This PR improves the listing of Vmware VMs for KVM migration tool to reduce the timeout observed while listing VMs on a datacenter. The improvement is performed in two stages:

First stage: Datacenter VMs listing -> use thin listing of VMs instead of obtaining all its disks and NICs (Invoke the listVmwareDcVms API setting the datacentername parameter only)
Second stage: When selecting a VM to import -> perform the full information retrieval of the selected VM including its disks and NICs information (Invoke the listVmwareDcVms API setting the datacentername, hostname and virtualmachinename parameter)

It was originally added by PR: #9925 but reverted as it caused unintended regression affecting the VM console access (#10634).

Fixes: #9782

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
build/CI
test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

Tested against Vmware vCenter datacenter with 200+ VMs, listing times reduced significantly from 30 seconds to <1 second for the entire datacenter VMs

Same command time difference against the same vCenter: (time cmk list vmwaredcvms vcenter=XXXX datacentername=XXXX username=XXXX password=XXXX page=1 pagesize=10 filter=name)

Before:

real	0m35.755s
user	0m0.087s
sys	0m0.016s

After:

real	0m0.468s
user	0m0.076s
sys	0m0.013s

nvazquez · 2025-04-24T09:56:30Z

@blueorangutan package

blueorangutan · 2025-04-24T09:58:04Z

@nvazquez a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

codecov · 2025-04-24T10:03:28Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 4.28%. Comparing base (55c8138) to head (456f3ac).
Report is 20 commits behind head on 4.19.

❗ There is a different number of reports uploaded between BASE (55c8138) and HEAD (456f3ac). Click for more details.

HEAD has 1 upload less than BASE

Flag BASE (55c8138) HEAD (456f3ac)

unittests 1 0

Additional details and impacted files

@@              Coverage Diff              @@
##               4.19   #10770       +/-   ##
=============================================
- Coverage     15.16%    4.28%   -10.89%     
=============================================
  Files          5412      371     -5041     
  Lines        475033    29724   -445309     
  Branches      57963     5215    -52748     
=============================================
- Hits          72048     1274    -70774     
+ Misses       394930    28305   -366625     
+ Partials       8055      145     -7910

Flag	Coverage Δ
uitests	`4.28% <ø> (-0.02%)`	⬇️
unittests	`?`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

DaanHoogland

clgtm

blueorangutan · 2025-04-24T11:19:05Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 13166

nvazquez · 2025-04-24T11:48:03Z

@blueorangutan test ol8 vmware-70u3

blueorangutan · 2025-04-24T11:50:03Z

@nvazquez a [SL] Trillian-Jenkins test job (ol8 mgmt + vmware-70u3) has been kicked to run smoke tests

.../vmware/src/main/java/org/apache/cloudstack/api/command/admin/zone/ListVmwareDcHostsCmd.java

blueorangutan · 2025-04-25T10:37:55Z

[SF] Trillian test result (tid-13127)
Environment: vmware-70u3 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 79615 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10770-t13127-vmware-70u3.zip
Smoke tests completed. 127 look OK, 6 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_role_account_acls	`Failure`	3.97	test_dynamicroles.py
test_role_account_acls_multiple_mgmt_servers	`Failure`	3.33	test_dynamicroles.py
test_01_events_resource	`Error`	355.18	test_events_resource.py
test_04_deploy_vm_for_other_user_and_test_vm_operations	`Error`	122.44	test_network_permissions.py
test_01_restore_vm	`Error`	3607.45	test_restore_vm.py
test_02_restore_vm_allocated_root	`Error`	10.30	test_restore_vm.py
test_12_destroy_cpvm	`Error`	6.36	test_ssvm.py
test_01_deploy_vm_on_specific_host	`Error`	19.06	test_vm_deployment_planner.py
test_02_deploy_vm_on_specific_cluster	`Error`	3607.06	test_vm_deployment_planner.py
test_03_deploy_vm_on_specific_pod	`Error`	2.57	test_vm_deployment_planner.py
test_04_deploy_vm_on_host_override_pod_and_cluster	`Error`	4.56	test_vm_deployment_planner.py
test_05_deploy_vm_on_cluster_override_pod	`Error`	15.80	test_vm_deployment_planner.py

nvazquez · 2025-05-05T21:04:44Z

@blueorangutan package

blueorangutan · 2025-05-05T21:06:04Z

@nvazquez a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-05-05T22:02:48Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 13265

nvazquez · 2025-05-05T22:23:50Z

@blueorangutan test ol8 vmware-70u3

blueorangutan · 2025-05-05T22:26:03Z

@nvazquez a [SL] Trillian-Jenkins test job (ol8 mgmt + vmware-70u3) has been kicked to run smoke tests

DaanHoogland

clgg,

DaanHoogland · 2025-05-06T06:30:19Z

vmware-base/src/main/java/com/cloud/hypervisor/vmware/mo/BaseMO.java

+                powerState == VirtualMachinePowerState.POWERED_OFF ? UnmanagedInstanceTO.PowerState.PowerOff : UnmanagedInstanceTO.PowerState.PowerUnknown;
+    }
+
+    protected List<UnmanagedInstanceTO> convertVmsObjectContentsToUnmanagedInstances(List<ObjectContent> ocs, String keyword) throws Exception {


complexity of this method is a bit high, would be nice to modularise it a bit.

Thanks @DaanHoogland, fixed

DaanHoogland · 2025-05-06T11:25:00Z

as this functionality was implemented in 4.19.2 (but with side effects) and thus would be a regression if not merged, I marked this as BLOCKER.

blueorangutan · 2025-05-06T12:55:44Z

[SF] Trillian test result (tid-13205)
Environment: vmware-70u3 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 49843 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10770-t13205-vmware-70u3.zip
Smoke tests completed. 133 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

rohityadavcloud

LGTM didn't test it though

NuxRo

LGTM, the speed differences are fantastic, good job!

nvazquez · 2025-05-12T15:49:49Z

@blueorangutan package

blueorangutan · 2025-05-12T15:52:03Z

@nvazquez a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-05-12T17:03:32Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 13348

nvazquez · 2025-05-12T20:58:58Z

@blueorangutan package

blueorangutan · 2025-05-12T21:00:03Z

@nvazquez a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-05-12T22:06:39Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 13352

nvazquez · 2025-05-12T23:02:58Z

@blueorangutan test ol8 vmware-70u3

blueorangutan · 2025-05-12T23:04:03Z

@nvazquez a [SL] Trillian-Jenkins test job (ol8 mgmt + vmware-70u3) has been kicked to run smoke tests

nvazquez added 3 commits April 24, 2025 06:49

Improve Vmware VMs retrieval

b770d27

Fix imports

8cac953

Fixes and improvements

16eab4d

nvazquez added type:bug component:vmware component:UI labels Apr 24, 2025

nvazquez added this to the 4.19.3 milestone Apr 24, 2025

nvazquez requested review from rohityadavcloud, kiranchavala, DaanHoogland, harikrishna-patnala, Pearl1594, sureshanaparti, vladimirpetrov, borisstoyanov and weizhouapache April 24, 2025 09:55

nvazquez linked an issue Apr 24, 2025 that may be closed by this pull request

[UI] Timeout listing Vmware Datacenter VMs for migration to KVM #9782

Open

nvazquez mentioned this pull request Apr 24, 2025

[UI] Timeout listing Vmware Datacenter VMs for migration to KVM #9782

Open

DaanHoogland approved these changes Apr 24, 2025

View reviewed changes

DaanHoogland added the status:needs-testing label Apr 24, 2025

Pearl1594 added this to ACS 4.20.1 Apr 24, 2025

Pearl1594 moved this to In Progress in ACS 4.20.1 Apr 24, 2025

sureshanaparti reviewed Apr 25, 2025

View reviewed changes

.../vmware/src/main/java/org/apache/cloudstack/api/command/admin/zone/ListVmwareDcHostsCmd.java Outdated Show resolved Hide resolved

Remove commented line

ab29fd8

DaanHoogland reviewed May 6, 2025

View reviewed changes

DaanHoogland added Severity:Critical Critical bug Severity:BLOCKER and removed Severity:Critical Critical bug labels May 6, 2025

rohityadavcloud approved these changes May 9, 2025

View reviewed changes

DaanHoogland assigned NuxRo and unassigned vladimirpetrov and borisstoyanov May 9, 2025

Refactor parameter names

065cd73

NuxRo approved these changes May 12, 2025

View reviewed changes

nvazquez added 2 commits May 12, 2025 11:38

Refactor method

4f83ca8

Add secure boot to the instance response

202f9d5

boring-cyborg bot added the component:api label May 12, 2025

Include boot type and boot mode

456f3ac

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Vmware] Improve listing of Vmware Datacenter VMs for migration to KVM #10770

[Vmware] Improve listing of Vmware Datacenter VMs for migration to KVM #10770

nvazquez commented Apr 24, 2025 •

edited

Loading

nvazquez commented Apr 24, 2025

blueorangutan commented Apr 24, 2025

codecov bot commented Apr 24, 2025 •

edited

Loading

DaanHoogland left a comment

blueorangutan commented Apr 24, 2025

nvazquez commented Apr 24, 2025

blueorangutan commented Apr 24, 2025

blueorangutan commented Apr 25, 2025

nvazquez commented May 5, 2025

blueorangutan commented May 5, 2025

blueorangutan commented May 5, 2025

nvazquez commented May 5, 2025

blueorangutan commented May 5, 2025

DaanHoogland left a comment

DaanHoogland May 6, 2025

nvazquez May 12, 2025

DaanHoogland commented May 6, 2025

blueorangutan commented May 6, 2025

rohityadavcloud left a comment

NuxRo left a comment

nvazquez commented May 12, 2025

blueorangutan commented May 12, 2025

blueorangutan commented May 12, 2025

nvazquez commented May 12, 2025

blueorangutan commented May 12, 2025

blueorangutan commented May 12, 2025

nvazquez commented May 12, 2025

blueorangutan commented May 12, 2025

[Vmware] Improve listing of Vmware Datacenter VMs for migration to KVM #10770

Are you sure you want to change the base?

[Vmware] Improve listing of Vmware Datacenter VMs for migration to KVM #10770

Conversation

nvazquez commented Apr 24, 2025 • edited Loading

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

nvazquez commented Apr 24, 2025

blueorangutan commented Apr 24, 2025

codecov bot commented Apr 24, 2025 • edited Loading

Codecov Report

DaanHoogland left a comment

Choose a reason for hiding this comment

blueorangutan commented Apr 24, 2025

nvazquez commented Apr 24, 2025

blueorangutan commented Apr 24, 2025

blueorangutan commented Apr 25, 2025

nvazquez commented May 5, 2025

blueorangutan commented May 5, 2025

blueorangutan commented May 5, 2025

nvazquez commented May 5, 2025

blueorangutan commented May 5, 2025

DaanHoogland left a comment

Choose a reason for hiding this comment

DaanHoogland May 6, 2025

Choose a reason for hiding this comment

nvazquez May 12, 2025

Choose a reason for hiding this comment

DaanHoogland commented May 6, 2025

blueorangutan commented May 6, 2025

rohityadavcloud left a comment

Choose a reason for hiding this comment

NuxRo left a comment

Choose a reason for hiding this comment

nvazquez commented May 12, 2025

blueorangutan commented May 12, 2025

blueorangutan commented May 12, 2025

nvazquez commented May 12, 2025

blueorangutan commented May 12, 2025

blueorangutan commented May 12, 2025

nvazquez commented May 12, 2025

blueorangutan commented May 12, 2025

nvazquez commented Apr 24, 2025 •

edited

Loading

codecov bot commented Apr 24, 2025 •

edited

Loading